Drill-Based Fitness Functions for Learning Extending Match-Based Training for Robot Soccer Agents

Some domains, like robot soccer, are difficult for agents to learn in using direct statistical and reinforcement learning techniques. However, agents often have a goal or purpose, which gives them a natural basis for reinforcement learning in the form of a fitness function. Such a fitness function allows the task of learning to be accomplished through optimization of the fitness function in the parameter space of parametrized agents. Unfortunately, such natural fitness functions are usually very general, leaving optimization subject to the curse of dimensionality and pay no attention to the shape of the fitness landscape they produce. Additionally, in multi-agent domains like robot soccer, the natural fitness function might not lead to a natural order on agents or teams of agents, which complicates optimization further. This thesis attempts to address these issues in the robot soccer domain by introducing alternative fitness functions that can be used to extend a natural fitness function, a match-based tournament. The alternative fitness functions considered are pig-in-the-middle training, attacking scenarios and defensive scenarios, all of which evaluate subsets of the parameters of agents. In addition, some of these fitness functions do lead to a natural order on teams of agents and others serve to smooth out the fitness landscape. An extended fitness function is an extension of the natural fitness function with a combination of alternative fitness functions. To evaluate the effectiveness of extended fitness functions, a genetic algorithm equipped with an extended fitness function is compared with a genetic algorithm equipped with a standard fitness function. Results are inconclusive since neither algorithm was able to converge in the time available. However, both algorithms lead to quick improvement over a hand-tuned team. Furthermore, the theoretical basis for the extension of natural fitness functions is persuasive, suggesting that a different experimental approach could lead to more conclusive results.